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Abstract 

We consider the reconstruction problem in compressed sensing in which the observations are recorded in a finite 
number of bits. They may thus contain quantization errors (from being rounded to the nearest representable value) and 
saturation errors (from being outside the range of representable values). Our formulation has an objective of weighted 
i2-£i type, along with constraints that account explicitly for quantization and saturation errors, and is solved with 
an augmented Lagrangian method. We prove a consistency result for the recovered solution, stronger than those that 
have appeared to date in the literature, showing in particular that asymptotic consistency can be obtained without 
oversampling. We present extensive computational comparisons with formulations proposed previously, and variants 
thereof. 

Keywords: compressive sensing, signal reconstruction, quantization, optimization. 

1 Introduction 

This paper considers a compressive sensing (CS) system in which the measurements are represented by a finite number 
of bits, which we denote by B. By defining a quantization interval A > 0, and setting G := 2 B_1 A, we obtain the 
following values for representable measurements: 

A 3A A A A 

-G+-,-G + — .,G--. (1) 

We assume in our model that actual measurements are recorded by rounding to the nearest value in this set. The 
recorded observations thus contain (a) quantization errors, resulting from rounding of the true observation to the 
nearest represented number, and (b) saturation errors, when the true observation lies beyond the range of represented 
values, n amely, \—G + 4-, G — 4-1- This setup is seen in some compressive sensing hardware archite ctures (see, for 



example. lLaska et al.L 120071: iTropp et al.1 120091: iRomberd, 120091: iTropp et aUl2006tlDuarte et all 120081) . 

Given a sensing matrix $ £ R Mx N and the unknown vector x, the true observations (without noise) would be $>x. 
We denote the recorded observations by the vector y £ R M , whose components take on the values in (HJ. We partition 
$ into the following three submatrices: 

• The saturation parts $_ and $ + , which correspond to those recorded measurements that are represented by 
— G + A/2 or G — A/2, respectively — the two extreme values in £[). We denote the number of rows in these 
two matrices combined by M. 

• The unsaturated part $ £ R MxN , which corresponds to the measurements that are rounded to non-extreme 
representable values. 



In some existing analyses (ICandes et al.L l2006t iJacques et al.L 1201 II) . the quantization errors are treated as a ran 



dom variables following an i.i.d. uniform distribution in the range [— — ,—]. This assumption makes sense in many 
situations (for example, image processing, audio/video processing), particularly when the quantization interval A is 
tiny. However, the assumption of a uniform distribution may not be appropriate when A is large, or when an inap- 
propriate choice of saturation level G is made. In this paper, we assume a slightly weaker condition, namely, that the 
quantization errors for non-saturated measurements are independent random variable with zero expectation. 
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The state-of-the-art formulation to this problem (see lLaska et aUl201 ll) is to combine the basis pursuit model with 
saturation constraints, as follows: 



mm x i 



(2a) 



s.t. \\$x - y\\' 2 < e 2 A 2 

$+x > (G — A)l 
< (A-G)l, 



(4) 

(+ saturation) 
(— saturation) 



(2b) 
(2c) 
(2d) 



where 1 is a column vector with all entries equal to 1 and y is the quantized subvector of the observation vector y that 
corresponds to the unsaturated measurements. It ha s been shown that the estimation error arising from the formulation 
(ffji is bounded by O(eA) in the £2 norm sense (see lLaska et all 1201 ltlCandesll2008tlJacques et all 1201 11) . 

The paper proposes a robust model that replaces d2bl by adding the least square loss in the objective and adds an 
^oo constraint: 



min — \\$>x 
x 2 



s.t. H&c-jflloo < A/2 

$>+x > (G - A)l 
< (A-G)l. 



(+ saturation) 
(— saturation) 



(3a) 

(3b) 
(3c) 
(3d) 



The £oo constraint arises immediately from the observation that the quantization error is bounded by A/2. This 
constraint may reduce the feasible region for the recovery problem while retaining feasibility of the true solution x*, 
thus promoting more robust signal recovery. From the viewpoint of optimization, the constraint d2bb plays the same 
role as the least-square loss term in the objective d3al l. when the values of e and A are related appropriately. However, 
it will become clear from our analysis that the least-squares loss can lead a tighter bound than using d2bb . 

The analysis in this paper shows that solutions obtained from this formulation are, in the worst case, better than 
the state-of-the-art model (O, and also better t han the mode l in wh ich only the £oo constraint d3bb are applied (in 
place of the £2 constraint d2bl), as suggested by Jacques et alj ( 201 1 ). More important, when the number M of unsat- 
urated measurements goes to infinity faster than S'log(iV) (where S is the sparsity number — the number of nonzero 
components in x*), the estimation error for the solution of (0) vanishes with high probability. By comparison, the 
state-of-the-art model (f2]i does not guarantee improvement given more measurements. Although lJacques et alj (1201 ll) 
show that the estimation error can be eliminated only using the £ x constraint, their "oversampling" condition is much 
stronger than our requirement. 

We apply the alternating direction method of multipliers (ADMM) (see lEckstein and Bertsekas , 1992tlBovd et al 



|20 1 lb to solve (0. The computational results reported in Section|4]compare the solution properties for (f3]) to those for 
d2J and other formulations in which some of the constraints are omitted. 



1.1 Related Work 



There have b een several re c ent wo rks on CS with quantization and saturation. lLaska et al.l (1201 11) propose the for- 
mulation (fJJ. Ijacques et al. ( 2011 ) replace the £2 constraint d2bl ) by an £ p constraint (2 < p < 00) to han dle the 
oversampling case, and show that values p greater than 2 are advantageous. The model of lZvmnis et al. I d2010l) allows 
Gaussian noise in the measurements before quantization, and solves the resulting formulation with an £\ -regularized 
maximum likeliho od formulation. T he average distortion introduced by scalar, vector, and entropy coded quantization 
of CS is studied bv lDai et alj d201ll) . 



The ext reme case of 1-bit CS (in which o nly the sign of the observation is recorded) has been studied bv lGupta et al 



J2010I) and lBoufounos and Baraniukl d2008f) . In the latter paper, the £\ norm objective is minimized on the unit ball, 
with a sign consistency constraint. The former paper proposes two algorithms that need no more than logiV) mea- 
surements to recover the unknown support of the true signal, but they cannot recover the magnitudes of the nonzeros 
reliably. 
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1.2 Notation 



• j denoting the £2 norm. We use x* for the true signal, 
as the difference between them. As mentioned above, S 



We use || • ||p to denote the £ p norm, where 1 < p < 00, with 
x as the estimated signal (the solution of ([3J, and h = x — x* 
denotes the number of nonzero elements of x* . 

For any z € W N , we use z, to denote the ith component and zt to denote the subvector corresponding to index set 
T C {1, 2, N}. Similarly, we use $t to denote the column submatrix of $ consisting of the columns indexed by 
T. The cardinality of T is denoted by \T\. 

In discussing the dimensions of the problem and how they are related to each other in the limit (as N and AI both 
approach 00), we make use of order notation. If a and f3 are both positive quantities that depend on the dimensions, we 
write a = 0(/3) if a can be bounded by a fixed multiple of (3 for all sufficiently large dimensions. We write a = o(/3) 
if for any positive constant <j> > 0, we have a < <fif3 for all sufficiently large dimensions. We write a = Q(/3) if both 
a = 0{(3) and (3 = 0(a). 

The projection onto the norm ball with the radius A is 

Vooix, A) := sign(ir) min(|x|, A) 

where denotes componentwise multiplication and sign(x) is the sign vector of x. (The ith entry of sign(aj) is 1, —1, 
or depending on whether Xi is positive, negative, or zero, respectively.) 

The indicator function In( - ) for a set IT is defined to be on II and 00 otherwise. 

Let 



$ = 



i>+ 



and $ = 



The maximum column norm in $ is denoted by / max , that is, 



max • lTlclX 

ie{l,2,...,JV} 



We define the following quantities associated with a matrix 'J with N columns: 



\^Th T \\l 



\r\<k,hem N \\hr\ 



and 



p„(k, \f0 := max 
p \T\<k,hm N 



\* T h T \\l 



(4) 



(5) 



(6) 



(7) 



Sometimes we use following notations for short: 



Pp 


(k) 


= Pp 


(A,*), 


Pp"(k) 


:=p+(*,$) 


Pp 


(k) 


= Pp 


(*,*), 




:=p+(fe,#) 


Pp 


(k) 


= Pp 


(*,*), 




:=p+(fe,#) 



Note that the £2 norm is used in each denominator, regardless of p. 
Finally, we denote max{z, 0} as (z)+ for short. 



1.3 Organization 

The ADMM optimization framework for solving (f3]l is discussed in Section|2] Section|3]analyzes the properties of the 
solution of (fj) in the worst case and compares with existing results. The numerical simulation is reported in Section|4] 
and some conclusions are offered in Section[5] Proofs appear in the appendix. 
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2 Algorithm 

This section describes the ADMM algorithm for solving (01. For simpler notation, we combine the saturation con- 
straints as follows: 



x > 



(G-A)l 
(G-A)l 

where $ is defined in (|4]i and y is defined in an obvious way. To specify ADMM, we introduce some auxiliary variables 
and write (f3]l as follows. 

min — \\<frx — y\\ 2 + A||x||i 

s.t. u = — y 

v = $?x — y 
Moo < A/2 
v > 0. 

Introducing Lagrange multipliers a and f3 for the two equality constraints in (0, we write the augmented Lagrangian 
for this formulation, with prox parameter 9 > as follows: 

L A (x,u,v,z,w,a,p,ri,j) 

$.t + y) + (/?, v 



(a, m 



-\\u — $x - 
2" 



y\\ 2 



-\\v - $x 
2" 



HI«||=o<A 



/2 (m) +I t ,> (w) 



(9) 



At each iteration of ADMM, we optimize this function with respect to the primal variables u and v in turn, then update 
the dual variables a and /3 in a manner similar to gradient descent. The parameter 9 may then be increased before 
proceeding to the next iteration. 

We summarize the ADMM algorithm in Algorithm Q] 



Algorithm 1 ADMM for © 

Require: <l, y, i>, y, A, K, and x; 

1: Initialize 6* > 0, a = 0, /3 = 0, u = Qx — y, and i; = &x — y; 

2: for k = : X do 

3: u «— argmiri,, : £,4(2:, u, w, a,j3,) = Voc(&x — y — a/9, A/2); 

4: w <— argmin„, : La(x, u, v, a, j3) = max($i — y — f3/9, 0); 

5: a; <— argmin^ : La(x,u,v,cx, /?); 

6: a «- a + 0(u — $x + J/); 

7: /3 4- ft + 0(v - $.t + y); 

8: Possibly increase 9; 

9: if stopping criteria is satisfied then 

10: break 

ii: end if 

12: end for 



The updates in Steps 3 and 4 have closed-form solutions, as shown. The function to be minimized in Step 5 consists 
of an 1 term in conjunction with a quadratic term in x. Many algorithms can be applied t o solve this prob lem, e.g., 
the SpaRS A algor ithm proposed bvlWright et al.l (12009b . the accelerated first order method ( iNesterovt 120071) . and the 
FISTA algorithm ( Beck a nd Teboulle, 200%. The update strategy for 9 in Step 7 is flexible. We use the following 
simple and useful scheme from lHe et al. I d2000b and lBovd et all J2OIII) : 



9t if ||r|| > n\ 
9 It if ||r|| < /i| 
6 otherwise, 



(10) 
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where r and d denote the primal and dual residual errors respectively, specifically. 

and d = 



u — $.t + y 
v — $w + y 



$ (X - Xl as t) 
$(£ - Xl as t) 



where .T] ast denotes the previous value of x. The parameters fj, and r should be greater than 1; we used fx = 10 and 
t = 2. 

3 Analysis 

The section analyzes the properties of the solution obtained from our formulation (|3). In Subsection 13. 11 we obtain 
bounds on the norm of the difference h between the estimator x given by (O and the true signal x*. Our bounds require 
the true solution x* to be feasible for the formulation (01; we derive conditions that guarantee that this condition holds, 
with a specified probability. In Subsection 13.21 we estimate the constants that appear in our bounds under certain 
assumptions, including an assumption that the full sensing matrix $ is Gaussian. 
We formalize our assumption about quantization errors as follows. 

Assumption 1. The quantization errors — y)i, i = 1, 2, . . . , M are indepently distributed with expectation 0. 

3.1 Estimation Error Bounds 

The following error estimate is our main theorem, proved in the appendix. 

Theorem 1. Assume that the true signal x* satisfies 

||$ T ($x*-y)||oo<AA/2. (11) 

Let s and I be positive integers in the range 1,2, ... , N, and define 

B L :=6ViAA, (12a) 

Boo :=y/MA, (12b) 

i4o(*) -=P2 0> + *,*)- zVWApHs + % *) - P2(s + 2/, *)] (12c) 
Ai(*) :=4\4(8 + 2l,*)- (£(8 + 21,*)], (12d) 




''W' <12s) 

We have that for any T C {1, 2, N} with s = \T \, ifA (<f>) > 0, then 

\\h\\ <C Q (m L +C 1 ($)/Vl\\x* TS \\ 1 +C 2 ($)Bl /2 /sV 4 \\x* TS \\{ /2 , (13a) 

\\h\\ ^(^oo + d^MH^olK. (13b) 

Suppose that Assumption\l\holds, and let n £ (0, 1) be given. If we define A = \/ 2 log 2N/ 7r/ max in ®. wif/i 
probability at least P = 1 — n, the inequalities d 1 3ab fl«t/ d 1 3bb /10W. 



From the proof in the appendix, one can see that the estimation error bound (1 1 3ab is mainly determined by the 
least square term in the objective (f3~ab . whereas the estimation error bound (1 1 3bb arises from the constraint d3bl . 

If we take To as the support set of a;*, only the first term in d 1 3ab and d 1 3bb remains. It follows that \\h\\ < 
min{Co($)i?L, C2{&)B QO }. We will compare their values in next subsection. 
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3.2 Estimating the Constants 



Here we discuss the behaviors of the least square term and the 1^ constraints by comparing the values of Co 
and C , 2( ( E > )i?oo- In order to simplify the comparison, we make the following assumptions. 

(i) $ is a Gaussian random matrix, that is, each entry is i.i.d., drawn from a standard Gaussian distribution 7V(0, 1). 

(ii) the confidence level P = 1 — n is fixed. 

(iii) s and I are both equal to the sparsity number S. 

(iv) Slog(N) = o(M). 

(v) the saturation ratio \ '■= M/M is smaller than a small positive threshold that is defined in Theorem|2] 

(vi) To is taken as the support set of x* , so that x^c = 0. 

Note that (iii) and (iv) together imply that s = I = S < M, while (iv) implies that M = Q(M). 

The discussion following Theorem[2]in Appendix indicates that under these assumptions, the quantities defined in 
d!2et , dl2fl . (I12fl) . and (O satisfy the following estimates: 



C (#) = fi(l/M), Ci($) = fi(l), C 2 ($) = Q(1/VM), /, 



with high probability, for sufficiently high dimensions. Using the estimates in Theorem[2] we have from the definitions 
of Bl and B M in TheoremQ] with the setting of A from TheoremQ] that 



C ($)Bl = O 



C 2 ($)S 00 = O 



^Slog(N)f n 
M 



O 



ISlog(N) 



M 



MA 



M 



= 0(A) 



Combining the estimation error bounds (II 3ab and ( I13bb . we have 



< min{C (l>)BL, C 2 ($)S 00 } = min{0( y/Slog(N)/M ), 0(1)}A. 



(14a) 
(14b) 

(15) 



In the regime described by assumption (iv), Cq(Q)Bl will be asymptotically smaller than C l 2($)i?no- The bound 



in ( fT~5T > has size O U/ S\og(N)/ M A), consistent with the upper bound of the Dantzig selector (ICandes and Tao[ 120071) 
and the LASSO (IZhana I2009H H Recall that the estimation error of the state-of-the-art algorithm © for quantized 



compressed sensing i s 0(\\$>x* - y\\/vM) jjacques et alll201 lllLaska et ai],l201 lb . Since ||l>x* - y\\ = O(vMA) 
jjacques et al. . 201 lb . this estimate is consistent with the error that would be obtained if we imposed only the £oo con- 



straint d3bl in our formulation. Under the assumpt ion (iv), the e s timati on error for (O will vanish as the dimensions 
grow, with probability at least 1 — n. By contrast. IJacques et al.l d201 II) do not account for saturation in their formu- 
lation and show that the estimation error converges to using the £oo constraint only when oversampling happens (in 
particular, M > Ne s /S) and M —J- oo. Our formulation yields a much weaker condition on M than oversampling. 
For example, M = S\og 2 (N) would produce consistency in our formulation, but not in ©. 



4 Simulations 

This section compares results for five variant formulations. The first one is our formulation Q, which we refer to in 
the plots as LASSOoo . We also tried a variant in which the constraint (|3bl was omitted from (|3). The recovery 

1 Their bound is O ( \J S log(N)/M a) where a is the standard deviation of the observation noise which, in the classical setting for the Dantzig 
selector and the LASSO, is assumed to follow a Gaussian distribution. 
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performance for this variant was uniformly worse than for LASSOoo , so we do not show it in our figures. (It is, 
however, sometimes better than the formulations described below, and uniformly better than Dantzig .) 

The remaining four alternatives are based on the following model, in which the £2 norm of the residual appears in 
a constraint (rather than the objective) and a constraint of Dantzig type also appears: 



min 

X 






(16a) 


s.t. || da; - y 2 


<e 2 A 2 




(16b) 


H&C- #||oo 


< A/2 


(tco) 


(16c) 


$ T ($x-y)\\™ 


< AA/2 


(Dantzig) 


(16d) 


<l + a; 


>(G- 


A)l (+ saturation) 


(16e) 


<l_a; 


<(A- 


G)l. (— saturation) 


(16f) 



The four formulations are obtained by omitting certain constraints from this model. 

• Loo : an £ nr , constraint model that enforces d!6ct . ( 116e| ), and dl6f| l, but not dl6b| > or dl6d| ) (that it, the model with 



p = 00 in lJacques et al.l d2011l) but with the addition of saturation constraints) 

• L2 : an £2 constraint model (that is, the state-of-the-art model in (O dLaska et al. , 201 1 )) that enforces d!6bl ), 
d!6et . and dl6f1 l but not d!6cb or d!6dl ); 

• Dantzig : the Dantzig constraint algorithm with saturation constraints, that is, it enforces d!6d| ), d!6e| ), and dl6fl ) 
but not d 1 6bb or d!6cb ; 

• L2Dantzigoo : the full model defined by dT6b . 

Note that we use the same value of A in d!6db as in (O, since in both cases they lead to a constraint that the true signal 
x* satisfies \\& T ($x* — y)\\oo < AA/2 with a certain probability; see d!6db and dTTb . 

The synthetic data is generated as follows. The measurement matrix <E> € M. MxN is a Gaussian matrix, each entry 
being independently generated from Af(0, 1/R 2 ), for a given parameter R. The S nonzero elements of x* are in 
random locations and are drawn from independently from Af(0, 1). We use SNR = — 20 log 10 (||a; — x*||/||x*||) as 
the error metric, where x is the signal recovered from each of the formulations under consideration. Given values of 
saturation parameter G and number of bits B, the interval A is defined accordingly as A = 2 B ~ 1 G. All experiments 
are repeated for 30 times; we report the average performance. 

We now describe how the bounds A for d3at and d 16db and e for d!6bb were chosen for these experiments. Es- 
sentially, e and A should be chosen so that the constraints d!6bl ) and d!6dl ) admit the true signal x* with a a high 
(specified) probability. There is a tradeoff between tightness of the error estimate and confidence. Larger values of 
e and A can give a more confident estimate, since the defined feasible region includes x* with a higher probability , 



while smaller values provide a tighter estimate. Although Lemma|2]suggests how to choose A and lJacques et alJ (1201 II) 
show how to determine e, the analysis it not tight, especially when M and TV are not particularly large. We use in- 
stead an approach based on simulation and on making the assumption (not used elsewhere in the analysis) that the 
non-saturated quantization errors = (<tx* — y)i are i.i.d. uniform in f7[_A/2,A/2]- (As noted earlier, this stronger 
assumption makes sense in some settings, and has been used in previous analyses.) We proceed by generating numer- 
ous independent samples of Z ~ ^[-a/2,a/2]- Given a confidence level 1 — 7r (for tt > 0), we set e to the value for 
which P(Z > eA) = tt is satisfied empirically. A similar technique is used to determine A. When we seek certainty 
(tt = 0, or confidence P = 100%), we set e and A according to the true solution x* , that is, e = \\<frx* — y||/A and 
A = 2||$ T ($a-*-y)|| 00 /A. 

To summarize the various parameters that are varied in our experiments: M and TV are dimensions of S is 
sparsity of solution x* , G is saturation level, B is number of bits, R is the inverse standard deviation of the elements 
of $, and P = 1 — tt denotes the confidence levels, expressed as a percentage. 

In Figure [U we fix the values of M, S, G, R, and P, choose two values of B: 3 and 5. Plots show the average 
SNRs (over 30 trials) of the solutions x recovered from the five models against the dimension N. In this and all 
subsequent figures, the saturation ratio is defined to be (M — M)/M, the fraction of extreme measurements. Our 
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LASSOoo formulation and the full model L2Dantzigoo give the best recovery performance for small TV, while for 
larger 7V, LASSOoo is roughly tied with the the L2 model. The Loo and Dantzig models have poorer performance — 
a pattern that we continue to observe in subsequent tests. 



M300-S1 0-B3-G4-R1 0-P1 00{Saturation=0.34) M300-S1 0-B5-G4-R1 0-P1 00{Saturation=0.24) 




Figure 1: Comparison among various models for fixed values M = 300, S = 10, G = 4, R = 10, and P = 100%, 
and two values of B (3 and 5, respectively). The graphs show dimension TV (horizontal axis) against SNR (vertical 
axis) for values of TV between 100 and 1000, averaged over 30 trials for each combination of parameters. 

Figure|2]fixes TV, M, B, G, R, and P, and plots SNR as a function of sparsity level S. For all models, the quality of 
reconstruction decreases rapidly with S. LASSOoo and L2Dantzigoo achieve the best results overall, but are roughly 
tied with the L2 model for all but the sparsest signals. The Loo model is competitive for very sparse signals, while the 
Dantzig model lags in performance. 

We now examine the effect of number of measurements 71/ on SNR. Figure [3] fixes TV, S, G, R, and P, and tries 
two values of B: 3 and 5, respectively. Figure [4] fixes B = 4, and allows TV to increase with M in the fixed ratio 5/4. 
These figures indicate that the LASSOoo and L2Dantzigoo models are again roughly tied with the L2 model when the 
number of measurements is limited. For larger M, our models have a slight advantage over the L2 and Loo models, 
which is more evident when the quantization intervals are smaller (that is, B = 4). Another point to note from Figured 
is that Loo outperforms L2 when both 71/ and TV are much larger than the sparsity S. 

In Figure[5]we examine the effect of the number of bits B on SNR, for fixed values of TV, M, S, G, R, and P. The 
fidelity of the solution from all models increases linearly with B, with the LASSOoo , L2Dantzigoo , and L2 models 
being slightly better than the alternatives. 

Next we examine the effect on SNR of the confidence level, for fixed values of TV, M, B, G, and R. In Figure|6] we 
set 71/ = 300 and plot results for two values of S: 5 and 15. In Figure|7] we use the same values of S, but set M = 150 
instead. Note first that the confidence level does not affect the solution of the Loo model, since this is a deterministic 
model, so the reconstruction errors are constant for this model. For the other models, we generally see degradation as 
confidence is higher, since the constraints ( I16bl ) and ( I16dl ) are looser, so the feasible point that minimizes the objective 
|| ■ ||i is further from the optimum x*. Again, we see a clear advantage for LASSOoo when the sparsity is low, M is 
larger, and the confidence level P is high. For less sparse solutions, the L2 and L2Dantzigoo models have similar or 
better performance. 

In Figure[8]we examine the effect of saturation bound G on SNR. We fix TV, 71/, 73, R, and P, and try two values of 
S: 5 and 10. A tradeoff is evident — the reconstruction performances are not monotonic with G. As G increases, the 
proportion of saturated measurements drops sharply, but the quantization interval also increases, degrading the quality 
of the measured observations. We again note a slight advantage for the LASSOoo and L2Dantzigoo models, with very 
similar performance by L2 when the oversampling is lower. 

In Figure|9] we fix TV, TV/, S, B, R, and tune the value of G to achieve specified saturation ratios of 2% and 10%. 
We plot SNR against the confidence level P, varied from 0% to 100%. Again, we see generally good performance 
from the LASSOoo and L2Dantzigoo models, with L2 being competitive for less sparse solutions. 



N500-M300-B4-G4-R1 0-P1 00 




S 



Figure 2: Comparison among various models for N = 500, M = 300, B = 4, G = 0.4, i? = 10, and P = 100%. The 
graph shows sparsity level S (horizontal axis) plotted against SNR (vertical axis), averaged over 30 trials. 



N500-S5-B3-G4-R1 0-P1 00(Saturation=0.1 8) N500-S5-B5-G4-R1 0-P1 00(Saturation=0.097) 




Figure 3: Comparison among various models for fixed values N = 500, S = 5, G = 0.4, R = 15, and P = 100%, and 
two values of B (3 and 5). The graphs show the number of measurements M (horizontal axis) against SNR (vertical 
axis) for values of M between 20 and 300, averaged over 30 trials for each combination of parameters. 
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Figure 4: Comparison among various models for fixed ratio N/M = 5/4, and fixed values S = 10, B = 4, G = 0.4, 
R = 15, and P = 100%. The graph shows the number of measurements M (horizontal axis) against SNR (vertical 
axis) for values of M between 100 and 1680, averaged over 30 trials for each combination of parameters. 




Figure 5: Comparison among various models for fixed values N = 500, M = 300, S = 10, G = 0.4, R = 10, and 
P = 100%. This graph shows the bit number B (horizontal axis) against SNR (vertical axis), averaged over 30 trials. 
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N400-M300-S5-B4-G4-R1 5(Saturation=0.021 ) N400-M300-S1 5-B4-G4-R1 5(Saturation=0.1 7) 




confidence confidence 



Figure 6: Comparison among various models for fixed values N = 400, M = 300, B = 4, G — 0.4, and R = 15, and 
sparsity levels 5 = 5 and S = 15. The graphs show saturation bound G (horizontal axis) against SNR (vertical axis) 
for values of P between 0.0001 and 0.99, averaged over 30 trials for each combination of parameters. 



N400-M1 50-S5-B4-G4-R1 5(Saturation=0.021 ) N400-M1 50-S1 5-B4-G4-R1 5(Saturation=0.1 7) 




confidence confidence 



Figure 7: Comparison among various models for fixed values TV = 400, M = 150, B = 4, G = 0.4, and R = 15, 
and sparsity levels S = 5 and S = 15. The graphs show confidence P (horizontal axis) against SNR (vertical axis) for 
values of P between 0.0001 and 0.99, averaged over 30 trials for each combination of parameters. 
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N500-M1 50-S5-B4-R1 0-P1 00 N500-M1 50-S1 0-B4-R1 0-P1 00 




Figure 8: Comparison among various models for fixed values of N — 500, M = 150, B = 4, R = 15, P = 100%, 
and two values of S: 5 and 10. The graphs show confidence P (horizontal axis) against SNR (left vertical axis) and 
saturation ratio (right vertical axis), averaged over 30 trials for each combination of parameters. 



N500-M1 50-S5-B4-R1 5-Sat2(G=0.31 ) N500-M1 50-S5-B4-R1 5-Sat1 0(G=0.22) 




confidence confidence 



Figure 9: Comparison among various models for fixed values of N = 500, M = 150, S = 5, B = 4, R = 15, and 
two values of saturation ratio: 2% and 10%, which are achieved by tuning the value of G. The graphs show confidence 
P (horizontal axis) against SNR (vertical axis), averaged over 30 trials for each combination of parameters. 
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Summarizing, we note the following points. 



( a) Our proposed LASSOoo formulation gives either best or equal -best reconstruction performance in most regimes, 
with a more marked advantage when the signal is highly sparse and the number of samples is higher. 

(b) The L2 model has similar performance to the full model, and is even slightly better than our model for less 
sparse signal s with fewer measu rements, since it is not sensitive to the measurement number as the upper bound 



suggested bv lLaska et al.l (1201 lb . Although the inequality in <JT5j also indicates the estimate error by our model 
is bounded by a constant due to the £ x constraint, the error bound determined by the constraint is not so 
tight as the £2 constraint in general. This fact is evident when we compare the the Loo model with the L2 model. 

(c) The Loo model performs well (and is competitive with the others) when the number of unsaturated measure- 
ments is relatively large. 

(d) The L2Dantzigoo model is competitive with LASSOoo if e and A can be determined from the true signal x*. 
Otherwise, LASSOoo is more robust to choices of these parameters that do not require knowledge of the true 
signals, especially if one desires a high confidence level. 



5 Conclusion 

We have analyzed a formulation of the reconstruction problem from compressed sensing in which the measurements 
are quantized to a finite number of possible values. Our formulation uses an objective of £2-^1 type, along with explicit 
constraints that restrict the individual quantization errors to known intervals. We obtain bounds on the estimation error, 
and estimate these bounds for the case in which the sensing matrix is Gaussian. Finally, we prove the practical utility 
of our formulation by comparing with an approach that has been proposed previously, along with some variations on 
this approach that attempt to distil the relative importance of different constraints in the formulation. 
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Appendix 



This section contains the proof to TheoremQ] developed via a number of technical lemmas. At the end, we state and 
prove a result (Theorem|2]i concerning high-probability estimates of the bounds under additional assumptions on the 
sensing matrix <J>. 

The proof of Theorem Q] essentially follows the standar d analysis procedure in compressive sen si ng. Some sim - 
ilar lemmas and proof s can be found in IBickel et al. I d2009h : ICandes and Taol (Eo07h : ICandesI d2008h : IZhanj (120091) : 
Liu et al.l (120101, 1201 21) . For completeness, we include all proofs in the following discussion. 

Let To be any subset of {1, 2, N}, with s :— \Tq\. Given the error vector h = x — x*, divide the complementary 
index set Tq := {1, 2, N}\To into a group of subsets T^'s (j = 1,2,..., J), without intersection, such that T\ 
indicates the index set of the largest I entries of Iit? , T2 contains the next-largest / entries of hr c , and so forth! 



Lemma 1. We have 



\$h\ 



< A. 



(17) 



2 The last subset may contain fewer than I elements. 
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Proof. From Ob} , and invoking feasibility of x and x*, we obtain 

Halloo = - x*)||oo < -y\\oo + ||&c* - y|U < A. 

□ 



Lemma 2. Suppose that Assumption\l\holds. Given ir £ (0, 1), f/ie choice A = *J 2 log (2N/ 7r)/ max ensures that the 
true signal x* satisfies dl 11 1, f/iaf is 

||* T (*a:* — sOHoo < AA/2 

vvi'f/i probability at least 1 — 7T. 

Proof. Define the random variable = $^($a;* — y) = wnere £ = •■•;?m] is defined in an obvious 

way. (Note that ||.2f||oo = ||$ T ($x* - y)||oo-) Since E(Zj) = (from Assumption!!} and all $ij&'s are in the range 
[— A/2, $jjA/2], we use the Hoeffding inequality to obtain 

r(Zj > AA/2) =P(Zj- -E(Z j ) > AA/2) 

M 

=nJ2^ - > xa/2) 

i=l 

-2(AA/2) 2 
<exp v ' ' 



-A 2 

: exp 



2V.$ 2 



< exp ■ 



A 2 



9f2 ' 

max 

which implies (using the union bound) that 

-A 2 

P(|Zj| > AA/2) < 2 exp 



J max 

-A 2 

>P(||^|| 00 = max|Zj| > AA/2) < 2iVexp- 



2 



where the last line follows by setting A to the prescribed value. This completes the proof. □ 
Lemma 3. We have 

j 

ll/nsiHElIM ^ IIMiM. 

where Tqi = To U Ti. 

Proof. First, we have for any j > 1 that 

l|/^ +1 || 2 < i\\h Tj+1 \L < KlHWx/i) 2 = \\h Tj \\i/i, 

because the largest value in |/it,- +i | cannot exceed the average value of the components of \hrA- It follows that 

j J-i 

\\h TSl \\ < E HM < E IIHIKM < ll^iliM- 

J=2 3=1 

□ 
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Similar claims or inequalities to Lemma|3]can be found in IZhandd2009l) :l ICandes and Taol(l2007l) : lLiu et al. (2010). 
Lemma 4. Assume that (fTTb holds. We have 

\\h TS \\i< 311^111+41^. Hi, (18a) 
\\h\\ < y/T+9^i\\hr 01 \\ +4||^||iM. (18b) 



Proof. Since x is the solution of (01, we have 

> \\\*x - y|| 2 - ill*** - yj| 2 + AA(||x||i - ||x*||i) 

> /i T $ T ($x* -y) + AA(||x||i - ||aj*||i) (by convexity of (l/2)||$x - y\\ 2 ) 
= h T $ T {$x* -y) + AA(||fi To ||i - IIztoIIi + \\xt S \\i - p T(f Ik) 

> -||fc|| 1 ||* r (* a: * - y)\\oo + AACPtJ! - H^JK + pr.ll! - P T .||i) 

> -||/i||iAA/2 + AAtprJx - p T J|i + \\£t S \\i + \\x* T o\\i - 2||o;?j||i) (from ©) 

> -dl^Tolli + ||/it H|i)AA/2 + AA(-||/*t„||i + IKHIi -2p T .||i) 

> ^AAll^lli - §AA||/i To ||i - 2AA||^.||i. 



It follows that 3 1 1 /; To 1 1 + 4||.x T c 1 1 1 > | h T g \ | i , proving (I18al 
The second inequality ( I18bb is from 



< II^To! II 2 + \\h TS \\i/l (from Lemma© 

< \\h Tol \\ 2 + (3\\h To \\i +4p T e|| 1 ) 2 /; (from W) 
<!|^ 01 || 2 + (3Vi||^To 1 ||+4||.T^|| 1 ) 2 // 
= (1 +9 S //)||/ lToi || 2 + 24vtyl||kr bl ||||a%|| 1 + 1611^^. ||?/Z 

v/1 + 9 S /;||/it 01 || +4!|x^|| 1 /\//] 2 . 

Lemma 5. Assume that ( II lb holds. For any matrix 'J w/f/j TV columns, and s,l < N, we have 

ll^f^AoWll^f-^iWII/iToJHI^IUM, 
where Aq(^) and A(^f) are defined in (I12cb a«c/ ( 1 12db respectively. 
Proof. For any j > 2, we have 

1 £oi £oi 2 J J J 1 
II^T 01 ||||^T,|! 



| *r 01 h Toi 1 1 1 /ir 01 1 1 + * Tj h Tj 1 1 1 h Tj 1 1 1 1 2 - 1 1 * Toi h Tal 1 1 1 h Tai 1 1 - * Tj /^T, / 1 1 h T] \ \ \ 



^Toi/II^Toil 
H/ll^ll 



[*T 01 : ] 



^T 01 /||/lT 01 || 
-H/IIHII 



< |(2p+(s + 20-2p2(s + 20) 



□ 



-(p+(s + 20-p 2 (« + 2Z)). 



(19) 
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The inequality above follows from the definitions (O and (0, and the fact that fact that /it i/||/it i|| an d hx /\\hx 
are i^-unit vectors, so that 



h T m /\\h Tm 
_ h T J\\hT s \ 

Considering the left side of the claimed inequality, we have 



^T 01 /||/lT 01 | 

-hTj\\h Tj \ 



2. 



m T c h T c I 



> \\*T M 



J">2 



> p 2 -(,s + OII^ToJI 2 - (Pi(s + 2l) - pz(s + 2l))\\h Toi \\ IIHH (fromlHD) 

> P2 {s + l)\\h Tal \\ 2 - {pt(s + 2l)-P2 (s + 20)||frT 01 ||||/*T=||lM (from LemmaEI) 

> pz(s + l)\\h Tol \\ 2 - (p+(s + 21) - pz(s + 20)||/ir ol ||(3||/iTol|iM + 4|jx^ \\ jVl) (from W) 

> (p2(s + l)-iV^l(pt(s + 2l)-p^( S + 2l))) \\h Tm \\ 2 - 

4(p+(s + 2l)-p^( S + 2l))\\x* TS \\ 1 \\h Tol \\/Vl (using \\h To \\i < V^\\h To \\ < V^\\h Toi \\) 

> A (*)\\h Toi \\ 2 - A 1 (*)\\hT 01 \\\\xT S \\i/Sl, 

which completes the proof. □ 
Lemma 6. Assume that ( fTTT i holds. We have 



\\$h\\ 2 K^XAWhWx < 6^XA\\h To 



(20a) 
(20b) 



\\$h\\ 2 <MA 2 , 
Proof. Denote the feasible region of (fTTT i as 

F := {x | $.t - y > 0, \\$x - < A/2}. 
Since x is the optimal solution to (01, we have the optimality condition: 

$ T ($i -y) + AAd||i||i n -N F (x) ^ 

where Np(x) denotes the normal cone of F at the point x and is the subgradient of the function j|.||i at the 

point x. It is equivalent to say that there exist g £ d\\x\\ i and n € Np(x) such that 



It follows that 



$ T (<M — y) + XAg + n = 0. 

<l T <i/i + l> T (l>a;* -y) + XAg + n = 
» h T $ T 3>h + h T $ T (3>x* -y) + XAh T g + h T n = 
» \\$h\\ 2 = -h T $ T {$x* -y)- XAh T g - h T n 

» \\§h\\ 2 < -h T § T ($x* -y)- XAh T g (using x* e F and h T n > 0) 
» ||*fc|| 2 < ||/i||i||$ T (*a:' - y)|U + AAl^lKHslU. 
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From 1 1 g | |oo < 1 and (fTTT i. we obtain 

\\$h\\ 2 < AA||fc||i/2 + AA||ft||i 

= §AA||&|| 1 

= 2 AA dl^ol|i + \\h TS \\i) 

< |aA(4||/i To || 1 +4||x^||i) (from (USD 

< 6^XA\\h To \\ +6AA||xtc||i, 

which proves the first inequality. 



From (TP71i, the second inequality is obtained by \\$h\\ 2 < a/A/||<I>/i||oo < A/A 2 . □ 



Proof of Theorem [T] 

Proof. First, assume that (fTTb holds. Take = $ in Lemma|5]and apply (I20at . We have 

^o(*)ll^r 01 || a -(A 1 (*)M)||x^ i ||i||hi bl || 
<II^H 2 

<6v^AA||&r J+6AA||:c?..||i 
=B L \\h Tal \\ + (B L /^- s )\\x* TS \\ 1 . 

If follows that 

A ($)||^ 01 || 2 - ((A^/Smx^Jt + B^hToA < Bl/V^IIi- (2D 

Using Ao($) > (which is assumed in the statement of the theorem), we recall that for a quadratic inequality 
ax 2 — bx < c with a,b,c > 0, one has 

b + yjb 2 + 4ac 2b + y/4ac b fc 

x< < = - + \~- (22) 

la la a \ a 

Hence d2ll implies that 



1 / - r \ B l\\x* t 4i 

hT " ^ T7^ 4 i $ )M ^ l i +Bl) + \ " V 



^ , lb* || 1+ ^ II-* ||Va 



By invoking dl8bt , we prove dl3at by 

\\h\\ < y/T+g^iWknA + (4/^)||^ Hi 



M*) V M$) ) KllT » lll/ 1 y ^o($) " To111 
= c q ($)b l + c.mxyyVi + c^B^iWx^/s^). 

Next we prove (II 3bb . Taking ^ = <t in Lemma[5]and applying d20bb , we have 

M*)\\hbif - (M*)/yft)\\xh M\ h T 01 \\ < \m\ 2 <MA 2 = Bl. 
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Using ( 122b again, one has 

\\h T J\<^(\\x* TS \\i/Vi) + ^^. 

By invoking dl8bt , we have 

\\h\\ <Vl+Wl\\hT 01 \\ + (4M)||x%||i 



/ yi+9VUi($)\ , /- l + 9£/Z R 
- 4 + 7717 ^ i/vi + W— — Boo, 

proving (|13bt . 

Note that all claims hold under the assumption that ( ITTb is satisfied. Since Lemma [2] shows that ( fTTT ) holds with 
probability at least 1 — it with taking A = log(2iV/7r)/ max , we conclude that all claims hold with the same 
probability. □ 

High- Probability Estimates of the Estimation Error 

For use in these results, we define the quantity 

X := M/M = (M - M) /M, (23) 
which is the fraction of saturated measurements. 

Theorem 2. Assume <f> £ M. MxN to be a Gaussian random matrix, that is, each entry is i.i.d. and drawn from a 
standard Gaussian distribution AT (0,1). Let $ € M. MxN be the submatrix of $ taking M rows from $, with the 
remaining M rows being used to form the other submatrix $ G ~R MxN , as defined in (|4j. Then by choosing a 
threshold r sufficiently small, and assuming that \ satisfies the bound x(l — log x) < T > we have for any k > 1 shc/z 
f/iaf A: log iV = o(M) f/iaf, with probability larger than 1 — 0(exp(— fi(M))), the following estimates hold: 

pt(k)<^VM + o(VM), (24a) 
ft(fc)>^%/M-o(VIO, (24b) 



<^+o(VI), (24c) 



P2(fc)>^VM-o(\^). (24d) 



Proof. From the definition of p^(fc), we have 



max CTmax($T), 
|T|<fc,TC{l,2,...,Af} 



where <r max (<I>x) is the maximal singular value of $y. From lVershvninl (1201 ll Theorem 5.39), we have for any t > 
that 

<w($r) < VM + O(Vfc) + « 
with probability larger than 1 — 0(exp(— fl(t 2 )). Since the number of possible choices for T is 



< 



eiV 
~k 
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we have with probability at least 



( )0(cxp(-fi(t 2 ))) 



that 



\[pUk)= max cr max ($ T ) < v / M + 0(v / A?) + i. 

V ^ lTl<fc.T(--f1.2 N\ 



|T|<fe,Tc{l,2,...,JV} 

Taking i = VM/16, and noting that fe = o(M), we obtain the inequality ( 124al i. with probability at least 

1 - 0(exp(fclog(e7V/fc) - Q{t 2 )) 
= 1 - 0(exp (k \og(eN/k) - 0(M)) 
= 1 - 0(cxp (o(M) - 0(M)) 
> 1 - 0(exp(-fi(M))) 

The second inequality d24bt can be obtained similarly from 

min cr m in($T) < VM - 0(Vk) - t, 

|T|<Jfe,Tc{l,2,...,JV} 



where er m in(3?T) is the minimal singular value of $t- (We set t — v M/16 as above.) 
Next we prove (I24ct . We have 



pj[k) = max — — — [j— < max cr max ($ flT ), 
where i? C {1, 2, . . . , M} and T C {1, 2, . . . , N} are subsets of the row and column indices of <E> , respective l y, and 



§r.t is the submatrix of $ consisting of rows in R and columns in T. We now apply the result in I Vershyninl (1201 1 
Theorem 5.39) again: For any t > 0, we have 

fTmax($i?,T) < + O(Vfc) + * 

with probability larger than 1 — 0(cxp (— £l(i 2 ))). The number of possible choices for R is 

AA feM\ Al /e^ M 



y < I -jgr ) =[-) = exp(Mxlog(e/ X )) < exp(rM), 



so that the number of possible combinations for (R, T) is bounded as follows: 

< cxp (tM + k\og{eN/k)) , 



M\ (N 
M)\k 



We thus have 



v(y$(k) < \^+0(Vk) + tj 
>P( max a{<f> RT ) < +0(Vk) + t 

\\R\<M.\T\<k 



1 - O [exp (rM + jfc log(eJV/jfe) - ^(t 2 ))] 
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Taking t = V M/16, and noting again that fc = o(M), we obtain the inequality in (124c b . Working further on the 
probability bound, for this choice of t, we have 



1-0 



cxp (tM + fc log(eN/k) - fi(M)) 

= 1-0 [cxp (tM + fc log(eJV/Jfc) - fi(M ))] 
= 1 - 0(exp(-0(M))), 

where the first equality follows from M = (1 — x)-^ ar, d f° r the second equality we assume that r is chosen small 
enough to ensure that the fi(AT) term in the exponent dominates the tM term. 

A similar procedure can be used to prove (I24dl ). □ 

We finally derive estimates of Cb($), Ci(<&), (^("J"), and / max , that are used in the discussion at the end of 
Section[3] 

From Theorem|2l we have that under assumptions (iii), (iv), and (v) the quantity A\ ($) defined in (I12db is bounded 
as follows: 



= 4 Jp+( S + 21) + Jp 2 "( S +2J) + 21) - Jfe{s + 21) 



< 4(2VM + o(VM))(-VM + o(VM)) 
8 

= M + o(M) = ft(M). 
Using s = Z, the quantity of Ao^) defined in d 12ct is bounded as follows: 

A ($)=p 2 -( S + 0-^^i($) 

> — M - o(M) - - o(M) 

= - o(M) 
= Q(M), 

for all sufficiently large dimensions and small saturation ratio x, since M = (1 — x)M. Using the estimates above for 
Aq($) and Ai($), together with s — I, in the definitions dl2el >, ( I12fl i. and ( 12g| i, we obtain 



c ,(i )g4+ W.(«L n(1)i 



C 2 (*) 



l + 9s/7 



as claimed. Finally, the estimate of / max can be estimated by /, 



= n(i/vM) 



p1(i)<£Vm + o(m) = o(Vm). 
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